This invention relates to sequence generators, and more specifically, to systems employing neural computation to develop sequences of output signal vectors.
Neural networks have been described by J. J. Hopfield in "Neurons With Graded Response Have Collective Computational Properties Like Those of Two-state Neurons"; Proc. Natl. Sci., USA Vol. 81, pp. 3088-3092, and by J. J. Hopfield and D. W. Tank in "`Neural` Computation of Decisions in Optimization Problems", Biological Cybernetics, Vol. 52, (1985), pp. 141-152; as well as in U.S. patent application Ser. No. 693,479 now U.S. Pat. No. 4,660,166 filed on behalf of J. J. Hopfield on Jan. 2, 1985, and U.S. patent application Ser. No. 795,789 now U.S. Pat. No. 4,719,591 filed on behalf of J. J. Hopfield and D. W. Tank on Nov. 7, 1985.
Basically, a neural network is a highly parallel computational circuit comprising a plurality of amplifiers, with each of the amplifiers feeding back its output signal to itself and all of the other amplifiers through conductance T.sub.ij. The T.sub.ij conductances (where T.sub.ij denotes the conductance between the output of amplifier j and the input of amplifier i) and the associated connections can be thought of as comprising a feedback network which has one output signal set and two input signal sets. The output signal set is applied to the amplifier inputs, one of the input signal sets is derived from the amplifier outputs, and the other input signal set is responsive to input stimuli applied to the neural network. As shown in the prior art, one can explicitly specify the values of the T.sub.ij conductances to achieve predetermined results, such as reaching different specified output states of the amplifier in response to different ranges of input stimuli. Also as described in the prior art, an input interconnection network may be interposed between the input stimuli and the second set of inputs of the feedback network. The input interconnection network permits manipulation of the expected input signals to corresponding signals that drive the feedback network and the amplifiers.
The neural network model described most extensively in the prior art is one with symmetric couplings, i.e., the connections between pairs of neurons satisfy the relationship T.sub.ij =T.sub.ji. The dynamics of such a network is relatively simple. The system moves in the direction of reducing a global energy function of the circuit, E, to states that are local minima of E, and once a local minimum is reached, the circuit remains at the stable state until perturbed by a sufficiently large input signal that moves the circuit to a different local minimum. The local minima may be thought of as the stored "memories" that are characterized by the vectors M.sup..mu.. An associative memory can be constructed with the Hopfield neural network by constructing the connection strengths T.sub.ij in accordance with the outer product, or Hebb, rule, to wit, by assigning ##EQU1## for j.noteq.i, and 0 otherwise.
A symmetric-coupling neural network thus does not provide temporal association. Asymmetric couplings, on the other hand, can generate sequential memories by adding weighted terms of the form M.sup..mu.+1 M.sup..mu. to the T terms in the feedback matrix. Each of these M.sup..mu.+1 M.sup..mu. terms is a projection of one "memory" to the next "memory". The problem with this technique for generating sequences is that only sequences of limited length are achieved because the state of the network becomes progressively mixed among memories comprising the sequence when long or cyclic sequences are used. More specifically, the essential difficulty in generating sequences through modifications of the feedback matrix is that the network does not become stationary in any one memory before transition to the next memory begins.
It is an object of this invention to provide a structure that simply and efficiently permits the realization of any desired sequence developed by neural networks.